Reputation Network Analysis for Email Filtering

In addition to traditional spam detection applications, new methods of filtering messages – including whitelist and social network based filters – are being investigated to further improve on mail sorting and classification. In this paper, we present an email scoring mechanism based on a social network augmented with reputation ratings. We present an algorithm for inferring reputation ratings between individuals and demonstrate through experiments that it is accurate based on current data. We then integrate those scores into a mail application, TrustMail, to show how they may be used in combination with other techniques for sorting and filtering mail. Introduction and Previous Work The fact that spam has become such a ubiquitous problem with email has lead to much research and development of algorithms that try to identify spam and prevent it from even reaching the user’s mailbox. Many of those techniques have been highly successful, catching and filtering the vast majority of Spam messages that a person receives. Though work still continues to refine these methods, some focus has shifted to new mechanisms for blocking unwanted messages and highlighting good, or valid, messages. “Whitelist” filters are one of these methods. In these systems, users create a list of approved addresses from which they will accept messages. Any whitelisted messages are delivered to the user’s inbox, and all others are filtered into a low-priority folder. These systems do not claim that all of the filtered messages will be spam, but rather that a whitelist makes the inbox more usable by only showing messages that are definitely not spam. Though whitelists are nearly 100% effective at blocking unwanted email, there are two major problems cited with them. Firstly, there is an extra burden placed on the user to maintain a whitelist, and secondly, valid emails will almost certainly be filtered into the low-priority mailbox. If that box contains a lot of spam, the valid messages will be especially difficult to find. Other approaches have used social networks for message filtering. In [1] Boykin and Roychowdhury create a social network from the messages that a user has received. Using the structural properties of social networks, particularly the propensity for local clustering, messages are identified as spam, valid, or unknown based on clustering thresholds. Their method is able to classify about 50% of a user’s email into the spam or valid categories, leaving 50% to be filtered by other techniques. Our approach takes some of the basic premises of whitelisting and social network based filtering and extends them. Unlike Boykin and Roychowdhury’s technique that builds a social network from the user’s own email folders, our technique uses a network that connects users. In our system, users assign a “reputation” or “trust” score to people they know. Their ratings indicate how a score of the email, and connects them to other users who, in turn, have their own set of ratings. The result is a large reputation network connecting thousands of users. Using a user’s personal view of the network, we apply a recursive algorithm to infer a reputation score for the sender of a message. That score is shown next to the messages in the inbox, and messages can be sorted according to this value. This works like a whitelist in that users can assign high reputation ratings to the people who would normally appear on a whitelist, and see that high score in their inbox. The scoring system preserves the whitelist benefit of making the inbox more usable by making “good” messages prominent. The added benefit is that scores will also appear next to messages from people with whom the user has never has contact before. That is because, if they are connected through a mutual acquaintance in the reputation network, we can infer a rating. This diminishes some of the problems with whitelists because, since scores are inferred instead of taken directly from a list, fewer valid messages will be filtered into a low-priority mail folder. Though some burden for creating an initial set of reputation ratings does fall on the user, it is possible to rate fewer people and rely on the inferred ratings. In the rest of this paper, we will describe mechanisms for creating and using reputation networks, the algorithms for inferring reputation ratings within the networks, and present TrustMail, our prototype application that uses the inferred values. Creating the Reputation Network The algorithms we use for calculating the reputation scores that will appear next to email messages are based on large, connected reputation networks. Unlike Boykin and Rodyhowdhury’s technique that builds a social network from the user’s own email folders, our technique uses a distributed, web based social network, and infers a reputation rating from one user to another by considering the paths between them. Each person in the network makes his or her own set of ratings. Individuals are connected to each person they rated. The neighbors, in turn, are connected to all of the people they rate. The result is a large, interconnected network of users. Figure 1 illustrates how two people, labeled A and B, become connected by their common neighbors. The only requirement for this system is that a network somehow be created where individuals assert their reputation ratings for one another. In the context of email, many of the large email providers can take advantage of their own population of users as a starting place. Service providers like Microsoft Networks (MSN), Yahoo! Mail, and America Online (AOL) have access to large numbers of users. Integrating reputation relationships into the other services offered to their users like Instant Messaging or chat would easily allow the members of a given service to create a foundational reputation network. To make an internet-wide reputation network that will allow ratings to be made for any individual is slightly more complicated. The network used for experiments in this paper is Semantic Web based. The benefits of the web in nearly every context is that individuals have control over their own data, and that data is maintained in a distributed way. While most social networks on the web currently are within closed websites, restricted only to members, the Semantic Web provides a method by which data can be stored anywhere and integrated through a common foundation. Figure 1: Building up a network through intermediate connections The Semantic Web, a World Wide Web Consortium (W3C) initiative, and its component languages of RDF, RDFS, and OWL, utilize existing web architecture and are designed to support exactly this type of distributed data management. Using the languages, users create ontologies with classes and properties, and then instances of those classes to describe data on the web. One of the most popular projects on the Semantic Web is the Friend-Of-A-Friend (FOAF) Project[3] an ontological vocabulary for describing people and their relationships. There are millions of FOAF files online – some created by individuals, and others output as a standardized way of sharing data from some centralized social network websites. Figure 2: The reputation network developed as part of the semantic web trust project at http://trust.mindswap.org. The Trust Project at http://trust.mindswap.org extends the FOAF vocabulary by providing a mechanism for describing the reputation relationships between people. It allows people to rate the reputation or trustworthiness of another person on a scale from 1 to 10 where 1 is a poor reputation (low trust) and 10 is a good reputation (high trust). This scale is intuitive for human users, but our algorithms can be applied to any scale of values. Our network contains nearly 1,500 people and is used as the foundation for this work. Figure 2 shows the current visualization of the reputation network. Algorithms for Inferring Reputation Between Individuals Perspective in Reputation Inference Algorithms In showing the potential utility of reputation networks, a good mechanism is required for inferring reputation ratings. A reputation inference algorithm – or reputation metric – is used with the reputation network and makes recommendations to one person (called the source) about the reputation of another person (the sink). The trust and reputation literature contains many different metrics, each designed and implemented for specific purposes. These metrics can all be categorized according to the perspective they use for making calculations. Global metrics calculate a single value for each entity in the network. Local metrics calculate a reputation rating for an individual in the network based on the ratings and preferences of the node that is the source of the query. In the global system, an entity will always have the same inferred rating. In the local system, an entity could be rated differently depending on which node the inference is made for. The choice of which method is best depends on the context. Global metrics can be highly effective in situations where the experiences of users are very similar. In a context where the users’ opinions about the same topic vary – like deciding whether or not an email is valid– a local metric may be more appropriate. Local reputation metrics use the neighbors and ratings of a particular node to make a reputation inference. The network in figure 1 illustrates a simple example of why the local metric is important in reputation networks. Since neither A nor B have a reputation rating for E, they must infer it. Each has a direct path through a single intermediate neighbor. Both A and B reputation their respective neighbors at a level 10 – the highest reputation rating. The neighbors, C and D, have very different opinions of E. Since A has high reputation in C, and C has low reputation in E, it should be more likely that A is recommended a low rating for E. On the other hand, B should receive a high reputation recommendation for E since B has high reputation for D, which in turn has high reputation for E. Figure 3: Depending on local opinions, inferred values can be very different. In this graph, Node A and Node B should receive very different opinions of Node E because of the opinions of their highly trusted neighbors differ. Accurate Metrics for Inferring Reputation To analyze how accurately reputation can be inferred over a network, we have created a simple metric[2]. The inferred rating from the source to the sink is given by a weighted average of the neighbors’ reputation ratings of the sink. Notationally, the reputation rating t from the source, i, to the sink, s, is written tis. If the source is directly connected to the sink, then no inference is necessary – the source already has a rating. If the two nodes are not directly connected, the source computes rating by calculating a weighted average of the reputation ratings returned for the sink by each of its n neighbors. Each of the source’s neighbors perform this same algorithm to find their ratings. 1 getRating(source, sink) 2 mark source as seen 3 if source has no rating for sink 4 denom = 0 5 num = 0 6 for each j in neighbors(source) 7 if j has not been seen 8 denom ++ 9 j2sink = min(rating(source,j),getRating(j,sink)) 10 num += rating(source,j) * j2sink 11 mark j unseen 12 rating(source,sink) = num/denom 13 return rating(source,sink) Formula (1) shows the concise representation of how tis is weighted. The condition in this formula ensures that the source will never trust the sink more than any intermediate node. € tis = (t js * tij ) if tij ≥ t js tij 2 if tij < t js 